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Abstract 
Recent trends in higher education have increased interest in improving accountability in U.S. higher 
education beyond current accreditation practices. Many proposed solutions include using quantitative- 
performance measures like graduation and default rates to assess performance, but questions remain 
about how such a system could fit-in with existing accreditation efforts. Using a unique dataset of 
accreditation actions, we examine the relationship between outcomes of the current accreditation 
system with those of a hypothetical quantitative evaluation system. We find that schools facing 
accreditation sanctions are, on average, low-performing on the quantitative outcomes. However, using 
prior accreditation actions to set quantitative-performance benchmarks results in a substantial portion of 
the higher education sector being implicated. These results suggest that quantitative-performance 
systems and qualitative accreditation efforts assess distinct, complementary types of institutional quality. 
We conclude with a consideration of how the two might be used together and a discussion of concerns 


about data and equity. 


Keywords: accreditation, postsecondary accountability, institutional performance, quantification, for- 
profit regulation 
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BOBBING FOR BAD APPLES: ACCREDITATION, QUANTITATIVE PERFORMANCE MEASURES, AND THE IDENTIFICATION OF LOW- 
PERFORMING COLLEGES 

On his visit to the United States in 1882, Oxford professor August Freeman recorded that, “One 
of the first things that strikes the stranger is the amazing number of universities and colleges...We can 
hardly be wrong in inferring that the degrees granted by some of these institutions cannot be worth very 
much... And perhaps we should not be wrong if we were to infer that it would be a gain if some of these 
degree-giving bodies were abolished or merged” (1883, p. 178). Freeman’s observations accurately 
reflect the historical truism that, in America, higher education has always been a growth industry. 
Attending this steady institutional growth has been a near constant concern for institutional quality and 
for where the authority to regulate that quality ought to reside. As Freeman observed, these issues ran 
headlong into American federalism. While “loath to say a word...against the powers of the several States,” 
Freeman offered, “it is surely not unreasonable to hint that the right of granting degrees should be 
assumed only by authority of the federal power. For a degree is surely a national thing” (1883, p. 179). 

This issue of how to balance the country’s uniquely open approach to regulating colleges with the 
need to protect students from lower-quality colleges all while respecting the rights of institutions, states, 
and the federal government has remained a considerable challenge for more than a century. Since the 
1970s, American higher education policy has relied on a—distinctly American—"triad” system of 
oversight and accountability involving voluntary accreditation, state government control through 
institutional charters and professional licensing boards, and federal government oversight exerted 
primarily through Title IV of the Higher Education Act (Harcleroad, 1980). While long considered 
imperfect, recent concerns about the rising price of college; increased student debt; reduced public 
funding of support for higher education; and concern that students are not learning useful skills have 
amplified concerns about whether the triad is robust enough to protect the interests of student 


consumers and American taxpayers (e.g. Flores, 2017; Kelchen, 2018). 


THE IDENTIFICATION OF LOW-PERFORMING COLLEGES 4 


Concern over lax enforcement—too many schools held to too little account—has resulted in 
criticism of accreditation standards and in calls for an increased focus on outcomes-based, market-driven 
accountability (e.g. Spellings, 2006). Following developments in a variety of private and public sectors 
including K-12 education, this “shift to outcomes” has resulted in a push to use more data and develop 
better metrics (e.g. Moynihan, 2008; Osborne & Gaebler, 1992; Trdhler, Meyer, Labaree, & Hutt, 2014). 
This information is intended to enable institutional accountability and to allow students and the broader 
public to make more informed decisions about the performance of schools receiving public money. One 
major challenge, however, is how to design an accountability system using quantitative-performance 
measures that responds to the limitations of the existing system and that credibly differentiates between 
good and bad performance in a way that is sensitive to the variety of institutional missions and 
constituencies of American higher education. 

In this article we aim to provide a clearer description of the challenge of designing a quantitative 
evaluation system by studying how a regulatory effort based on quantitative performance measures 
would differ from, and perhaps compliment, the current regulatory system. In particular, we take up the 
task of greatest concern to policymakers and the general public: How can quantitative measures be used 
to identify the “lowest performing” postsecondary institutions. 

One central problem in implementing a quantitative performance system is where to establish 
the thresholds for various outcomes below which an institution might be deemed “low performing.” 
Qualitative determinations like accreditation are subjective and susceptible to charges of inconsistency, 
but bright-line quantitative measures are susceptible to critiques that they reify arbitrary cut-points. We 
address this problem in our analysis by starting from a point that nearly every agrees on: current 
accountability efforts are too lenient in their assessment of college performance. Therefore, as one 
approach, we assume that colleges that have faced accreditation sanctions (e.g. warnings or loss of 


accreditation) are very likely to be low-performing. We can then use these accreditation judgements, 
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lenient though they may be, as a starting point for calibrating performance thresholds for a quantitative 
evaluation system—using the statistical profile of schools that faced accreditation actions to identify 
similar schools that did not. These thresholds give us a starting point to compare accreditation and 
quantitative-performance evaluation systems, and perhaps suggest ways the two systems might work in 
tandem to successfully identify low-performing schools. 

Using data on recent accreditation actions, and federal postsecondary performance data, we 
answer three main research questions: (1) How do schools that faced accreditation sanctions perform on 
various quantitative performance measures? (2) Using these quantitative performance measures, how 
many accredited schools perform worse than schools in accreditation trouble when compared to the 
lowest performing, highest performing, or average school that faced accreditation sanctions? And 
conversely (3) how many schools in accreditation trouble would be identified as low performing under an 
accountability system that classified schools according to their quantitative-performance outcomes? 

In answering these questions, this article makes several contributions to the literature on 
postsecondary accountability, the consequences of quantification, and the measurement of 
postsecondary quality. First, the few prior attempts to examine the work of accreditors have looked 
either at outcomes from a subset of all accreditors (e.g. Flores, 2016) or limited their analysis to 
accreditation actions contained in public data (e.g. Government Accounting Office, 2014), resulting in an 
incomplete picture of the operation of the current accreditation system. We improve on these prior 
efforts by compiling and using multiple sources of accreditation data to produce a more comprehensive 
database of accreditation actions for our analysis. Second, this more complete account provides a basis 
for comparing accreditor actions to quantitative measures of institutional performance to a degree of 
detail not possible in earlier work. Doing so provides an opportunity to compare the performance of the 
existing accreditation system (one that is likely to persist for the foreseeable future) to hypothetical 


designs for new quantitative accountability systems. This comparison offers fresh insight about the extent 
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to which accreditors and quantitative performance measures align and diverge in their assessment of 
institutional performance. A clear account of the respective capacities of these systems, like the one 
provided here, we believe must be the starting point for future examinations how to develop more robust 
and complimentary accountability systems. 

Accreditation and Quantitative Performance Measures 

Although the means of achieving their stated purposes are quite different, the current 
accreditation system and proposed quantitative-performance accountability systems have remarkably 
similar aims. As shown in Table 1, there is a great degree of overlap between the purposes of 
accreditation, and the stated aims, for example, of the Obama administration’s proposed, outcomes- 
based college rating system (CRS). Indeed, both systems aim to provide information for students about 
college quality, ensure that colleges meet particular standards, and spur college improvement (Hegji, 
2014, U.S. Department of Education, 2014). These similarities make comparisons between outcomes 
reached by accreditation agencies and outcome-based measures worthy of consideration. 

[ Insert Table 1 here ] 

Where the two systems differ, however, is in the way that they approach their assessment of 
colleges. Accreditation happens via qualitative self- and peer-assessment, while quantitative-performance 
accountability typically attaches monetary consequences to under-performance on numerical- 
performance metrics. These differences reflect the different institutional logics that shape the way actors 
think about and approach their work under each system. While both are grounded in a compliance-based 
logic of state monitoring, accreditation activities are characterized by the professional logics of peer- 
review, organizational learning, and improvement (Brown, 2017). Quantitative-performance measures do 
not incorporate these professional norms but instead emphasize the views of consumers and taxpayers 
through the market-based logic of performance and incentives (Burke 2005; Brown, 2017). To illustrate 


how these differing logics inform the design of their respective monitoring systems, consider that 
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accreditation process involves both the self-study and peer review of the professional administration of 
an institution. In contrast, quantitative measures like those in in the College Scorecard highlight the 
salaries of the school’s graduates and the percentage of students paying down their student loan debt. In 
the former, the emphasis is on standards maintained and enforced by people within the institutions. In 
the latter, the emphasis is on the market logic of consumer's return on investment. These varied logics 
have important implications for how each system responds to low-performing institutions. Understanding 
these differences is an important first step in understanding the results produced by each system. 
The Current Accreditation System 

College accreditation began in the early 1900s as an informal, voluntary system of establishing 
basic distinctions among institutions and codify basic standards for admission and completion. This 
accreditation was used as a way for institutions to qualify their professors for Andrew Carnegie’s lucrative 
faculty pension plan, administered by the Carnegie Foundation for the Advancement of Teaching 
(Rudolph, 1960/1990). While maintaining institutional and professional standards remained a goal of 
accreditation, following World War Il, the federal government, rather than develop its own quality- 
enforcement agency, began using accreditation as a way of ensuring the quality of institutions that 
received federal money (Harcleroad, 1980). Since the Higher Education Act of 1965, accreditation has 
been one of the criteria to determine an institutions eligibility for Title IV funding, primarily via student 
financial aid. Over time, the purposes of accreditation have evolved further as new institutional types, 
programs, and expectations have developed. The result is now an accreditation system that embodies the 
needs of the state to monitor institutions and of institutions to maintain of the professional norms of 
their field. 

In the accreditation system’s current form, the accreditation of whole institutions is performed 
by federally-recognized regional or national accrediting agencies. As a general rule, regional accrediting 


agencies typically accredit public and private institutions, while national agencies accredit for-profit and 
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career-oriented institutions (Eaton, 2015). In a typical accreditation review, institutions first conduct a 
self-assessment of their performance, which forms the basis for a site visit conducted by administrators 
and faculty members from peer institutions (Eaton, 2015). Institutions are assessed in several areas 
including fiscal and administrative capacity; facilities and equipment; faculty; curricula; and student 
achievement (20 U.S.C. § 1099b(a)(5)). 

A successful accreditation review typically results in 10 years of accreditation before another 
review is required (Kelchen, 2018). Less successful reviews may result in additional scrutiny or a shorter 
accreditation term. The most serious sanction an accrediting agency can take against a low-performing 
institution is to revoke its accreditation, meaning that the institution would no longer be eligible to 
receive federal Title IV funding meaning a student cannot receive federal financial aid funds if they attend 
that university. A Government Accountability Office (GAO) report found that, over a four-and-a-half-year 
period, 8 percent of institutions were cited for not meeting accreditor standards, while 1 percent had 
their accreditation revoked (GAO, 2014). Most of the terminations were for financial capability issues at 
non-degree granting institutions (GAO, 2014). Although, academic quality infractions were the most 
common at public and not-for-profit institutions, there is only weak evidence that accrediting bodies 
sanction schools with worse student outcomes (GAO, 2014). 

While many believe these outcomes reflect overly lenient approach to accreditors’ oversight 
responsibilities, the accreditation system with its structure of peer-assessment, sensitivity to individual 
school missions, and an ethos of supporting institutional improvement embodies a professional logic that 
does not support the strict sanctioning of low performance (Brown, 2017). Indeed, accreditors appear 
unwilling to resort to the “nuclear option” of eliminating an institution’s Title IV eligibility (The Triad, 
2013) and hesitate to issue minor sanctions in order to avoid potential lawsuits (Reauthorizing the Higher 
Education Act, 2015). Thus, although there is dissatisfaction with the rigor of accreditation—as Secretary 


Duncan put it, accreditors are “the watch dogs that don’t bite” (Stratford, 2015)—it may be that the 
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system is not currently designed to respond satisfactorily to current demands for more rigorous 
accountability. 
Quantitative-Performance Measures and Accountability 

Developing alongside the growing sense that accreditation agencies were not doing enough to 
regulate institutions in the face of rising college costs, state and federal governments increasingly 
expected institutions to produce information to allow for their evaluation by the public and to be held 
responsible for their organizational results (Burke, 2005; Zumeta, 1998). As in the private sector (e.g. 
Osborne & Graeber, 1992), the use of quantitative-performance measures was intended to direct 
institutional attention to important outcomes often through financial incentives and to direct public 
attention to institutions’ relative performance on these measures. Adoption of quantitative performance 
systems was often seen by lawmakers as a way to produce better outcomes and efficiency by introducing 
market logics to higher education oversight while preserving institutional autonomy and as a way to 
provide a more objective means of assessing performance and making consequential decisions (Porter, 
1996; Frederickson & Fredrickson, 2006; Moynihan, 2008). 

During the 1980s and 1990s, the vast majority of states adopted some form of higher education 
performance system involving either performance reporting, performance funding, or performance 
budgeting (McLendon, Hearn, & Deaton, 2006). This interest in performance metrics continued in the 
2000s as many states sought to retool their performance funding systems (Dougherty et al., 2014; 
Rabovsky, 2012; Tandberg & Hillman, 2014) and as major foundations launched efforts aimed at pushing 
states to improve their data practices and adopt performance funding plans (Complete College America, 
2009; see also Hearn, McLendon, & Mokher, 2008). 

The federal government, likewise, sought to raise the availability and visibility of school 
performance data over the last three decades by creating new data reporting requirements and new 


public-facing platforms for the public to access the collected information: In the 1990s the federal 
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government required schools to report cohort default rates; the Spellings Commission (2005) called for 
increased transparency and accountability for measures of institutional performance; and the Obama 
administration developed the College Scorecard and worked to design a college rating system that would, 
like state performance funding policies, tie federal financial backing to student outcomes (U.S. 
Department of Education, 2014). 

The increased use of performance metrics has raised questions about the nature of their 
influence on the systems they monitor. While there is strong evidence that the information provided 
about public sector institutions by performance metrics influences private individual behavior in the 
context of, for example, charitable giving (Figlio & Kenny, 2009), school choice (Hastings & Weinstein, 
2008), and health insurance selections (Dafny & Dranove, 2008), many scholars express concerns that the 
logic of performance management is inconsistent with democratic governance (e.g. Radin, 2006) and is 
incapable of adequately handling the multi-faceted missions and complex administrative structures of 
modern government (e.g. Frederickson & Frederickson, 2006). This limited ability to manage institutional 
complexity without reducing it to a single metric or organizational logic is a particular concern in the 
application of quantitative-performance metrics to institutions of higher education that have multiple and 
varied constituencies, multi-faceted missions, and complex governance structures (Burke 2005). The 
problem is further amplified by attempts to create explicit performance rules. Though the rules 
themselves may be designed to provide a more objective way of separating high and low-performing 
institutions, they may be open to the charge of unfairly privileging certain aspects of the institutional 
mission over others, of disadvantaging institutions for the populations they serve, or, more generally, of 
assigning arbitrary importance to specific performance thresholds. For instance, on-going debates about 
the development of the gainful employment rules reflect each of these concerns (Fain, 2013). 

A related set of concerns stems from the belief that performance metrics are not ineffectual, but 


rather that they work too well—inducing undesirable individual and institutional responses in an effort to 
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improve performance indicators (e.g. Campbell, 1976). For example, institutional responses to ranking 
systems have led schools to alter their organizational processes and values (e.g. Espeland & Sauder, 2016; 
Strathern, 1997) and, in the context of public K-12 education, school districts developed a wide variety of 
strategies to game accountability metrics associated with No Child Left Behind (e.g. Figlio & Loeb 2011). 
Given these prior experiences there are legitimate concerns that a similar response would occur in higher 
education if a full-blown quantitative evaluation system were ever implemented, especially if it were tied 
to federal funding (Deming & Figlio 2016). Evidence from the implementation of federal cohort default 
rate rules, tied like accreditation to Title IV eligibility, suggests that while overall rates are generally 
declining, schools may be achieving this result by advising students to engage in behavior that is 
beneficial from the perspective of the institution’s performance metrics but perhaps not in the student’s 
own best interest (Kelchen & Li, 2017). Likewise, responses to Obama’s proposed College Rating System 
were replete with such concerns (e.g. Field, 2014). 

Substitutes or Complements? 

Having considered the operation of the current accreditation system and the arguments and 
concerns related to a quantitative evaluation system, we now turn to the task of considering how they 
might be joined together within the existing accountability triad. Given that accreditation is the lynchpin 
of that system—state and federal policies are often built on top of accreditor determinations—we are 
interested in investigating how the results of an evaluation system based on quantitative performance 
measures from currently available data would compare with those derived from past accreditation 
actions. Though they are clearly different systems—drawing on different institutional logics and different 
data sources—comparing their respective evaluation of schools could shed light on whether the two 
systems work toward the same performance goals, but to different degrees, or if they judge institutions 
differently and so prompt thinking about how they might best be used in tandem to increase 


accountability pressure in American higher education. 
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For instance, based on the common belief that accreditors succeed in identifying low-performing 
schools but, bowing either to political, public, or membership pressure, do so far less often than they 
should, it is possible that the statistical profile of schools that had accreditation actions against them 
could be used to “benchmark” a quantitative performance evaluation system. That is, to use the 
quantitative profile of the schools that did receive accreditation sanction to identify statistically similar 
schools that, perhaps undeservedly, escaped accreditors notice. In this scenario, a quantitative 
performance system could be used to address the problem of the sleepy accreditation watchman. 
Likewise, it is possible that using available statistical data will identify sets of schools that are clear outliers 
in terms of their performance. In this scenario, the quantitative performance system could serve as a 
further tool of accreditors or as additional policy lever for lawmakers and regulators. 

In the work that follows, we aim to compare accreditor actions with school performance based 
on a variety of quantitative-performance measures in order to determine to what extent these 
assessments of institutional performance align, or—to the extent that they differ—the different types of 
colleges they identify as low performing. We focus on net price, graduation rates, and cohort default 
rates as our primary quantitative-performance measures of interest. These outcomes are common to 
policy conversations about postsecondary accountability, and most recently were included in the 
discussions of President Obama’s proposed college rating system (U.S. Department of Education, 2014). 

Data 
Accreditation Data 

Our primary source for data on the accreditation status of institutions is the Database of 
Accredited Postsecondary Institutions and Programs from the U.S. Department of Education.* This 
database, which we refer to as the “ED data,” comprises all accreditation actions reported to ED since the 
early 1900s and contains school-identifying variables, including the identification variable that allows the 


data to be liked with the Integrated Postsecondary Education Data System (IPEDS), as well as information 
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about the type of accreditation (institutional or programmatic), the accrediting agency, the last 
accrediting action taken by the agency, and the dates of accreditation. 

As a supplemental source of accreditation data, we also use data collected directly from 10 major 
regional and national accrediting bodies by the Center for American Progress (Flores, 2016). These data 
were collected between 2012 and 2015 directly from online, public notices of actions taken by each of 
the accrediting bodies. They provide a good complement to the ED data because, in many cases, they 
include actions that were not required to be reported to ED. Even though they do not cover actions taken 
by all accrediting bodies, these data add nuance not included in the full ED accreditation dataset. We 
refer to this dataset as the “CAP data.” Together, these two datasets represent the most complete 
accreditation data analyzed in scholarly research. 

College Scorecard Data 

We merge the accreditation data with data from the College Scorecard, which aggregates and 
generates data from several sources including IPEDS, the National Student Loan Data System, the Federal 
Student Aid Data Center, and other data sources from the Department of Treasury and ED. The scorecard 
data is useful for two purposes. First, it provides basic descriptive information about institutions, 
including their control and degree length, the size of their undergraduate enrollment, and the 
racial/ethnic composition of the student body. Second, it provides outcome data that have been 
suggested as measures by which institutions might be held accountable. These include graduation rates in 
150% of expected time’; three-year cohort default rates; and net price. The three-year cohort default 
rate gives the percent of borrowers who entered repayment on a Federal Stafford or Direct Stafford/Ford 
loan in a given year who then defaulted on their loans within the next three years. Net price gives the 
total cost of attendance minus the average amount of grant/scholarship aid received by students. 


Data Delimitations 
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We focus our analysis on the institutional accreditation of Title IV-level institutions (which may be 
multi-campus). Programmatic accreditation also exits, but it is typically granted to specific training 
programs such as nursing to indicate that they meet the standards of their industry. Our focus on 
institutional accreditation is consistent with discussions of outcomes-based postsecondary accountability 
that have focused on rating complete institutions. 

Further, we consider only degree-granting, two- and four-year colleges and universities. This is 
first because non-degree institutions are unique in form and function from two- and four-year degree 
granting colleges so it would not make sense to hold them accountable with the quantitative 
performance standards. Second, our data does not include the reasons for accreditor sanctions. Yet, 
when it comes to comparing accreditation actions or quantitative performance measures, we would 
prefer to focus on sanctions related to academic quality rather than financial capability. Given that we 
know that most financial capability sanctions occur among non-degree-granting colleges (GAO, 2014), we 
exclude them from our analysis. 

The ED and CAP data include multiple observations per year. We collapse the data into academic 
years to align it with the College Scorecard data. For the ED data, an academic year includes actions 
reported from August through June of the following year. Because of apparent reporting delays, June 
accreditation actions in the CAP data are aligned with the subsequent academic year in order to better 
match the ED data. We limit our analysis to actions taken between the 2011-12 and 2014-15 academic 
years reflecting the years for which we have both CAP and ED data and the availability of our outcomes 
measures. 

In order to allow for a clear comparison between schools that did and did not face accreditation 
action, the final data was reduced to one observation per institution. For institutions that faced 
accreditation sanction, we keep the observation for the academic year in which the accreditation action 


occurred. For institutions with no accreditation actions, we keep the observation from 2014-15. The 


THE IDENTIFICATION OF LOW-PERFORMING COLLEGES 15 


choice of 2014-15 rather than other years does not qualitatively affect our findings because institutional 
performance was relatively consistent in the four years of the data. 
Accreditation Actions 

We group institutions into four categories for our analysis. The first group consists of colleges 


vu 


who have had their institutional accreditation “withdrawn,” “terminated,” or are “no longer recognized” 
by their accrediting agency. Any school that has one of these as its last action is categorized as having lost 
its accreditation. In a few cases, there are schools that regain accreditation after appeal. We treat these 
schools as having lost their accreditation under the original action, even though the institution continues 
to operate. 

The second group, within the ED data, consists of institutions that are listed as under “probation.” 
These institutions offer an example of colleges that are performing poorly enough to be of concern to 
accreditors even if that performance is not enough to revoke accreditation While probation has long been 
a tool used by accrediting agencies to incentivize compliance with agency standards, it is only recorded as 
an accountability action in the ED data since 2012-13. Within the CAP data, we consider probation as part 
of a group of “major actions” that accreditation bodies take that are just short of revoking accreditation. 
Flores (2016) notes in describing the CAP data that these actions may vary in name ranging from “show 
cause” to “probation”, but they all represent an expression of serious concern on the part of the 
accrediting body. 

The third group, which is only available in the CAP data, describe “minor actions” taken against 
offending institutions. These include actions such as “warnings” and “heightened monitoring” that 
indicate concern for the performance of an institution, but not at a level that would warrant one of the 
major actions described above. 

Finally, we consider colleges in both datasets that are operating in good standing with their 


accreditors and who have no record of adverse accreditation action. 


THE IDENTIFICATION OF LOW-PERFORMING COLLEGES 16 


Missing Data 

Although reporting to IPEDS is mandatory for Title IV institutions, many schools do not submit 
more than just basic identifying information. For example, 15 percent of the graduation rate data are 
missing. The missing data problems are worse, perhaps predictably so, in schools that have lost their 
accreditation or are on probation. Thirty-two percent of these schools (in the ED data) are missing 
graduation rate data. We have no recourse to impute the missing data values for most of our variables 
because they are the primary outcome of our study. Instead we report the statistics for the observations 
for which we have data. We believe the available data still allow us to provide an illustration of colleges 
with varying accreditation status. 

Method 

Our method is primarily descriptive. We give descriptive statistics for each outcome of 150% 
graduation rate, 3-year cohort default rate, and net price. We also calculate regression-adjusted 
graduation and default rates. These provide expected graduation and default rates for each institution 
based on institutional type and student body composition. We create these by first calculating: 

Rate;; = Bo + B,perPell;; + Ba,perBlk;; + BsperHisp;; + ByperAsn;; + Bsenroll;; + Bgcontrol;; 
+ Brlevel;; + yj + &i; 

In other words, for each institution (/) in the year they are observed (/), we predict graduation (or 
default) rate, controlling for percent Pell recipients; percent Black, Hispanic, and Asian; total enrollment; 
institutional control (public, private, or for-profit); level (2- or four-year); and year fixed effects. This 
specification reflects known correlates of graduation and default rates (e.g. Hillman, 2014; Scott, Bailey, & 
Kienzl, 2006) and the fixed effects correct for varying conditions between years that might affect the 


graduation rates of all schools in that year. 
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We use the coefficients from this regression to predict expected graduation and default rates 
using an institution’s own characteristics, which we then compare to a school’s actual graduation rate and 
present as the percentage points an institution’s graduation or default rate was over/under its prediction. 

With these quantitative performance metrics, we then work to compare schools with 
accreditation actions taken against them and schools that perform poorly in the quantitative metrics. This 
occurs first by finding the worst, average, and best performance among the groups of schools with each 
accreditation action taken against them and finding how many accredited schools performed worse than 
those benchmarks. Finally, using the quantitative performance measures as our guide, we look for 
obvious cut points in the data that would include, for example, all colleges that lost their accreditation. 
Specifically, we look at how many of each type of institution that had accreditation actions taken against 
them perform relative to the top and bottom quartile of performance on our regression-adjusted 
measures, as well as how many schools underperformed relative to prediction. 

Results 

Basic descriptive statistics (means, column percent, and counts) for each of the analytical groups 
of colleges in each dataset are found in Table 2. In the ED data, accreditors took 79 actions against 
institutions. More of the probation actions reported were undertaken by regional accreditors, while 
national accreditors terminated twice as many accreditations as regional accreditors. Consistent with 
their respective roles, national accreditors took more actions against for-profit institutions, particularly 
two-year institutions, while regional accreditors mostly focused their actions on public and not-for-profit 
institutions. Generally, public institutions had the fewest actions taken against them. 

Colleges that lost their accreditation tended to have larger black student populations and fewer 
Hispanic and Asian student populations relative to schools that maintained their accreditation. Colleges 
that were put on probation tended to have a slightly higher percentage of Pell recipients, at least among 


regionally-accredited institutions. 


THE IDENTIFICATION OF LOW-PERFORMING COLLEGES 18 


[ Insert Table 2 here ] 

Similar patterns held in colleges described by the CAP data. In the wider array of accreditation 
actions recorded in the CAP data, there were 440 accreditation actions taken against institutions. 198 of 
these were minor, 211 were major, and 20 involved a termination of accreditation. Similar to the ED data, 
regional accreditors were much more likely to take minor actions than national accreditors (198 instances 
vs. 11 instances), but each type terminated the same number of accreditations. Further, nearly all of the 
actions taken by national accreditors were against for-profit institutions. Also parallel to the ED data, 
college with higher percentages of Black students were more likely to lose their accreditation, while 
institutions that had major actions taken against them had higher percentages of Pell recipients. 

We report the remainder of our results by institution level and control rather than by accreditor 
type because the categories are so closely aligned. 

Average Performance on Outcomes-Based Measures 

Table 3 summarizes how institutions in each of our analytical groups performed on average in our 
five college outcomes-based measures: average net price, graduation rate in 150 percent of expected 
time, performance relative to a regression-adjusted graduation rate, three-year cohort default rate, and 
performance relative to a regression-adjusted default rate. We present these measures separately for 
two- and four-year colleges, and by institutional control for each of our two data sources. 

[ Insert Table 2 here ] 

In the ED data, four-year colleges that lost their accreditation cost students about as much, on 
average, as schools that maintained their accreditation (roughly $19,000), while colleges that were put on 
probation cost less (roughly $16,500). In the CAP data, schools that lost their accreditation, at both the 
two- and four-year level were more expensive that schools that kept their accreditation. 

In terms of student outcomes, in both datasets, four-year institutions that had their accreditation 


terminated or had other actions taken against them had lower graduation rates than those who 
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maintained their accreditation. Four-year institutions in the ED data that lost their accreditation or were 
put on probation have lower graduation rates—37 percent and 31 percent respectively—relative to the 
average 48 percent six-year graduation rate at accredited four-year institutions. These disparities remain 
even after accounting for institution characteristics. Schools that lost their accreditation and probation 
schools fell between 12 and 13 percentage points below their expected rate. This trend was largely driven 
by private and for-profit colleges, which underperformed expected graduation rates regardless of 
accreditation action. The converse was true at two-year institutions. 

The opposite was true at two-year colleges, where colleges in accreditation trouble generally had 
higher graduation rates than those not in trouble in both datasets. However, at two-year colleges, the 
public and private institutions that lost their accreditation, actually over-performed their graduation 
expectations by an average of 15 percentage points in the ED data. 

With respect to default rates, four-year institutions in the ED data generally had higher default- 
rates if they were put on probation or lost their accreditation (13.7 and 12.8 percent) than if they kept 
their accreditation (8.9 percent). This higher rate could not be explained by the students those 
institutions served—four-year colleges in accreditation trouble had higher default rates than the 
regression-adjusted predictions by over 1.5 percentage points. The data was thinner at the two-year 
level, but where we do have data, we see that public-two-year colleges that lost their accreditation had 
lower default rates than those who kept it or were put on probation. This was also true, on average, for 
two-year colleges in the CAP data. 

With few exceptions related to low numbers of observations, the average performance of four- 
year colleges in both the CAP and ED data mostly align with the judgement of accreditors. However, these 
averages occlude the full distribution of institutional performance. They fail to reveal the extent to which 


there are accredited schools that, according to these measures, are indistinguishable from schools in 
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accreditation trouble. That is, whether accreditation actions can help benchmark outcomes-based 
measures. 

Table 3 also shows that, in addition to this concern about the distribution of performance, 
accreditation bodies appear to identify nearly the opposite schools than expected at the two-year level. 
Among two-year colleges in the ED data, schools that lost accreditation graduated their students at 
higher rates—although not when adjusting for the composition of the student body—and graduated 
students /ess likely to default on their loans than schools that maintained their accreditation. One 
explanation for this stark contrast with four-year institutions is that ED graduation rates essentially treat 
transfer students as drop-outs. For this reason, institutions that work to help their students transfer to 
four-year colleges will look like they are graduating fewer students than institutions with no similar 
transfer mission. Likewise, if students transfer from two- to four-year colleges, and take additional loans 
to help pay for the additional years of schooling, they will inherently be at a higher risk of defaulting on 
those loans, which may also explain why accredited two-year colleges have higher default rates—it may 
be precisely because these schools that are doing a good job of transferring students to four-year 
colleges. 

Thus, even before we consider the number of accredited schools not meeting the standards of 
schools that were in accreditation trouble, we can divergence between the results accreditation and 
outcomes-based systems. To wit, schools with generally “good” performance on graduation and default 
rate outcomes are treated differently by accreditors between two- and four- year colleges. This 
distinction reveals either important nuance that accreditors bring to their evaluations, or a seeming 
blindness to performance outcomes. 

Benchmarking Quantitative-Performance Measures. 
Thus far we have been examining how well schools that have had an accrediting action taken 


against them compare on average with those schools that remained in good standing with their 
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accreditors. If one challenge of developing a parallel system based on quantitative performance is how to 
establish the threshold of “low-performing,” then it is worth considering how many colleges have similar 
outcomes to schools in accreditation trouble. Ideally, we would hope that the accreditation actions allow 
us to home in on a statistical profile that could be rigorously and reliably monitored through an 
outcomes-based system. If on the other hand, lots of schools look statistically like those that faced 
accreditation sanctions, it might raise questions about the credibility of the benchmarks. That is, if too 
many schools are identified as “failing,” then the distinction ceases to carry meaning and its application to 
specific institutions ceases to be an effective mechanism for inducing change. 

To examine how many schools look statistically similar to those who had accreditation actions 
against them, in Table 4 (and 5), we compare the performance of four-year (two-year) institutions that 
maintained their accreditation to that of the lowest performing school that lost its accreditation that we 
observe for each outcome; the average performance of the schools we observe that lost their 
accreditation; and the highest performing school that lost its accreditation. We repeat these same three 
comparisons using schools given probation/major action and minor action rather than those that lost 
their accreditation. 

[ Insert Table 3 here] 

Even relative to the lowest performing school that lost its accreditation across each of the three 
outcomes, a substantial number of institutions that still have their accreditation in the ED data perform 
worse. For example, over six percent of accredited four-year colleges, roughly 151 institutions, have 
graduation rates lower than the single lowest graduation rate among the schools that lost their 
accreditation. The comparisons are even less flattering if we compare schools to the average 
performance of institutions that lost their accreditation. Anywhere from 24 to 31 percent of institutions 
with accreditation in good standing perform worse than the average school that lost its accreditation on 


graduation rate, graduation rate relative to predication, and default rate; and over half have higher net 
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prices. Finally, a substantial majority of accredited four-year colleges perform worse than the highest 
performing college that lost its accreditation across all four measures. Comparisons to schools on 
probation appear only slightly more favorable, with roughly one-in-five accredited colleges performing 
worse than the average school on probation across most measures. 

The type of school implicated by each comparison varies by the outcome used. Public institutions, 
predictably, are less likely to be labeled as low-performing under net-price based metrics. Public 
institutions are more likely than private institutions to be identified as low-performing using graduation 
rate alone, but are less likely to be labeled as such when judged relative to their regression-adjusted 
prediction. Meanwhile, private schools look better using raw default rates than they do with comparisons 
to predicted default rates. Four-year, for-profit fare the worst in almost every comparison. 

Similar relative patterns hold in the CAP data as in the ED data, though in most cases fewer 
accredited schools perform worse than the benchmarks set by schools with accreditation actions taken 
against them. However, all institutions in the CAP data have lower graduation performance, regardless of 
adjustment, than the schools given minor sanctions. Such high performance of minor action schools 
suggests that accreditation bodies might use this action as a way of sanctioning what are otherwise well- 
performing schools for minor evaluation problems. 

With respect to the two-year colleges in Table 5, for-profit colleges tended to perform better 
than public and private colleges on both graduation outcomes (for example, roughly 13 percent had 
lower graduation rates than the average graduation rate of a two-year college that lost accreditation, 
relative to 84 percent of public colleges). And depending on the comparison, private two-year colleges 
tended to perform well on each of the default-rate measures. 

[ Insert Table 5 here ] 
It may be desirable to hold schools accountable based on their performance on multiple 


measures. As an example of this, Figures 1 and 2 give scatterplots of graduation and default rate 
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performance for four-year (Figure 1) and two-year colleges (Figure 2) based on the control of the 
institution (public, private, or for-profit). We indicate on each plot where the performance on each 
outcome of the lowest-, average-, and highest-performing colleges at each level that lost their 
accreditation. Figure 1 shows, for example, that 4 colleges perform worse than the lowest performing 
four-year college that lost accreditation on both graduation and default rate measures. One of these 
schools is public while 3 are private. 430 four-year schools perform worse on both measures than the 
average four-year college that lost accreditation. Thus, even comparing accredited schools to schools that 
lost their accreditation across multiple measures at the same time marks a great deal of these schools as 
low-performing.? The same rough pattern is depicted in the CAP data, except that there is less variation in 
the default rates of schools in accreditation trouble in the CAP data than in the ED data (Appendix Figure 
Al). 

[ Insert Figure 1 here ] 

[ Insert Figure 2 here ] 

Figure 2 shows a stark divide on graduation rate measures between public-and for-profit two- 
year colleges. Most public two-year colleges have three-year graduation rates less than 45 percent, while 
for profit schools almost uniformly have graduation rates higher than 45 percent. This dichotomy means 
that when we look at the 868 accredited schools performing worse on both measures than the average 
two-year schools that lost their accreditation, the vast majority of them are public institutions. If one of 
the unspoken goals of postsecondary accountability is to crack down on for-profit community colleges, 
rating them on graduation rate and default rates, as currently measured, does not appear to be not the 
way to do it. Again, the CAP data shows a similar pattern, displayed in Appendix Figure A2. 

Accreditation Performance Relative to Quantitative Thresholds 
A slightly different way to consider the relative performance of the accreditation system to a quantitative 


system would be to consider how well quantitative measures distinguish between colleges using obvious or 
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natural cut points in the data rather than schools that lost accreditation. To illustrate this test, we compare 
schools on both regression-adjusted graduation and default rates. In Table 6, we give the percentage of schools in 
each of our analytical groups whose performance versus predicted graduation and default rates fell in the bottom 
25 percent of colleges according to their level (two- or four-year schools), those schools that underperformed 
relative to their prediction at all, and those who were in the top 25 percent of their comparison group—these are 
common data delineations, but we note there are no obvious clusters in the data that suggest natural alternative 
divisions. We would hope that in creating these broad evaluative categories the system could, at the very least, 
group those with accreditation actions against them in with the lowest category group. 

[ Insert Table 6 here ] 


Among the institutions in the ED data, roughly 25 percent of accredited two- and four-year schools fall in 


£ 


the top and bottom 25 percent of graduation and default rate performance relative to prediction, as would be 
expected. However, none of the cut points in either measure perform well at separating schools that had 
accreditation actions taken against them. They get the closest in identifying the few schools that lost their 
accreditation—for example, 80 percent of four-year colleges that lost their accreditation in the ED data, and 100 
percent of the ones in the CAP data, underperformed on the graduation measure. However, this measure is less 
good among two-year colleges, two-thirds of the colleges that lost accreditation in the ED data were in the top 25 
percent of all two-year colleges with respect to their graduation rate relative to prediction. Similarly, 36 percent 
of the colleges on probation in the ED data were in the top 25 percent of overall two-year college performance. 
Perhaps reflecting that the CAP data appears to capture a relatively lower performing set of 
institutions in general, a greater proportion of the CAP data schools in accreditation trouble 
underperform relative to prediction than in the full ED dataset. In fact, all of the (eight) four-year CAP 


schools that lost their accreditation were underperforming with respect to their graduation rates. 


However, the cut points did not distinguish well among colleges with other accreditation actions against 
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them. For example, over half of the CAP data four-year colleges with major actions taken against them, 
were in the top-25% of four-year colleges in terms of the default rate performance relative to prediction. 
Discussion 

We conducted this analysis to examine the extent to which either accreditation actions might be 
used to benchmark quantitative performance measures, or quantitative performance measures could be 
used to distinguish between institutions with accreditation actions taken against them. While these were 
each reasonable possibilities, our results suggest neither of these uses is likely to be fruitful in practice: 
the schools flagged by accreditors did not possess a unique statistical profile; and schools, far from 
coalescing into discrete performance groups, fell on a nearly continuous line on available outcome 
metrics. Even when using input-adjusted outcome measures to create broad categories of performance, 
we found that schools marked for sanction by accreditors would rarely have stood out in a quantitative- 
performance system. In other words, there was very little alignment between the work of accrediting 
bodies and quantitative measures of institutional performance. 

These findings underscore a major challenge in the design of quantitative accountability systems: 
calibrating the identification of low performing schools in justifiable ways. While quantitative systems are 
clearly helpful in removing the political pressure associated with the subjective identification of low- 
performing schools because they do so by objective rules, what Porter (1996) calls “mechanical 
objectivity, but how to set these rules remains unclear. In many critiques of accreditation actions there is 
an assumption of the overly-permissive watchman who is too frequently willing to look the other way 


£ 


either out of professional courtesy, public pressure, or fear of conflict or lawsuits. If this were the case, 


: 4 


then a quantitative system might be used to produce more reliable enforcement of these standards. But 
our results indicate that accreditors’ judgments are not easily captured or reproduced by available 


quantitative information. Indeed, our attempt to generalize on the basis of the statistical performance of 


schools identified for accreditation sanction led to a seeming over-identification of schools. This is 
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problematic as it diminishes the effectiveness of being identified as low-performing. As Figlio and Loeb 
(2011) explain in the context of NCLB potentially identifying a// schools as not meeting “adequate yearly 
progress,” “accountability systems that set standards such that a massive fraction of schools would likely 
fail may be perceived as incredible by educators who do not believe that central authorities would shut 
down schools...on a grand scale” (389)—indeed, the creation of a waiver policy was ultimately used to 
avert this outcome. 

The seeming inability of outcome measures to piggyback on the judgements of accreditors or to 
provide clear distinctions between higher- and lower-performing schools, suggests that the best way 
forward for the development of a future quantitative evaluation system may be the setting of bright-line 
thresholds. This is consistent with Deming and Figlio (2016)’s assessment of the lessons to be learned by 
American higher education from the decades of accountability policy enacted at the K-12 level. They 
argue that the best system design will seek to set a single, relatively permissive standard but will enforce 
it vigorously. Though any specific threshold may be subject to challenges that it has been “arbitrarily” set, 
as was recently the case with the gainful employment rule (Fain, 2013), such threshold measures have 
proved effective at generating institutional and market pressures. For instance, the introduction and slow 
ratcheting up of cohort default rate rules over several decades has led to a general decline in the overall 
rates of default (Kelchen, 2018). While these rules do not work perfectly—there is evidence that schools 
have attempted to game the system and rules can still change in response to political pressure—they 
have succeeded in weeding out some of the worst offending programs and spurred institutional change 
in others (Kelchen & Li, 2017). There is the additional benefit that debates about the “acceptable” 
performance threshold, squarely frame what is, ultimately, a normative question: when it comes to 
consumers and taxpayers, how much risk is too much? 

Beyond this threshold question, there are several additional difficult issues that any future 


quantitative system of accountability will need to address concerning issues of data quality, equity, and 
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how accreditation efforts might be used in quantitative-performance measures before it can be 
considered viable addition to our current system. The next step then is to think about how these systems 
complement each other. We identify several issues below that are key to making this work possible. 
Data Challenges 

A successful system to quantitative measurement requires the production of better, higher 
quality information about colleges. One remaining roadblock, evident in our analysis, to implementing 
any system based on quantitative-performance measures successfully is that existing data is currently too 
weak to support it. Some of these weaknesses, such as transfers counting against school graduation rates, 
are known by ED and presumably could be addressed (U.S. Department of Education, 2014). Indeed, 
recent changes to the graduation rate calculations have included more non-traditional student 
populations, but still struggle to account for students who transfer away from two-year colleges 
(Lederman, 2017). Of greater concern, however, is that even though graduation rate data has been 
collected and reported in IPEDS since the 2000-01 school year, it is missing entirely for nearly a quarter of 
the schools in our analysis. Three-year cohort default rate data, although only collected more recently, 
also shows similar rates of non-response. The missing data problems compound themselves when we 
consider that 40 percent of schools are missing either one of the graduation rate or default rate 
measures. Though a more robust system might extract higher rates of compliance, reporting this data is 
already required by law. As with accreditors who are limited in their ability to exert pressure short of the 
outright removal of accreditation, calibrating the appropriate level of sanction for non-data response may 
prove challenging. 
Equity Challenges 

There is a legitimate concern that the current accreditation system is too lax and may give a pass 
to some schools simply because of their historical reputation or their standing within a community even 


when the school is providing students with low-quality educational opportunities and taxpayers with high 
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risk educational investments. It makes sense, then, that we try to use a quantitative performance system 
to try and improve institutional performance. But one major concern for any quantitative evaluation of 
U.S. higher education is that it not unfairly penalize schools serving non-traditional students, especially 
those who struggle to access and afford higher education. Our analysis demonstrated that schools facing 
accreditation sanctions serve higher proportions of Black students than other schools. Additionally, no 
fewer than two million Pell recipients attend either two- or four-year colleges that perform worse than 
the average performance of schools that faced accreditation sanctions (authors’ calculations). 

These are large numbers and if an accountability system is to be successful, it must avoid two 
pitfalls with these students. First, it has to ensure that institutions are not incentivized to close their doors 
to these students, who typically have lower graduation rates regardless of where they attend, in an 
attempt to raise their ratings performance. Second, these (and all) students would need advice about 
where they can successfully enroll, if the schools they currently choose are so low-performing. 

There is the additional concern that the introduction of more robust systems based on 
quantitative performance may create a policy environment in which it becomes more challenging to have 
honest conversations about the structural reasons behind individual school statistics or that the values 
embedded in the evaluation system press schools to constrain or abandon parts of their mission. Our 
results suggest the possibility that these problems are not just theoretical. For instance, we found that 
that public and for-profit two-year schools performed fundamentally differently in terms of graduation 
and default rates. It is not hard to come up with reasons why this might be —different missions, 
stakeholders, opportunity structures, or levels of commitment to the public good. While proposed 
outcomes systems have tended to avoid the identification of specific categories of schools and have 
tended to remain silent on potential explanations for differences in school performance, it is not clear 
that this is wise. Consistent with the existing distinction between regional and national accreditors and 


with recent calls for the federal government to expand the number and character of authorized 
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accreditors to allow for greater innovation in educational programs (e.g. Kelly & James, 2014), it may 
make sense to segment any quantitative-performance evaluation system by institutional type. 
Complementary Systems 

Had our analysis revealed that the judgment of accreditors and the quantitative-performance 
variables had collectively homed in on a discrete set of schools, it would have been possible to conclude 
that these two systems—one expert, one statistical—had converged on a definition of school quality. In 
that case an argument could be made that an outcomes-based system provides a simpler, more cost- 
effective, and possibly more transparent method of applying accountability pressure. However, we find a 
divergence that suggests the statistical and peer-qualitative judgements are operationalizing different 
constructs of school quality. This would imply that it would be unwise to eliminate one system in lieu of 
the other—they each appear to offer an important perspective on institutional operation and 
performance. 

This distinctiveness suggests that quantitative-performance measures might be used to 
strengthen the qualitative assessments of accreditation bodies. If one of the qualities of the current 
accreditation system is its sensitivity to a wide array of institutional missions and contexts, then that 
judgement might be used to temper the bright-line system we discuss above. For example, accreditation 
teams could use discretion and expert judgment—for example, considering the populations a school 
serves or recent improvement efforts—to determine whether the performance institutions on either side 
of the bright-line cutoff warrants sanction. Quantitative outcome measures might also provide the 
opportunity to employ a more nuanced set of sanctions. Given the reluctance of accreditors to resort of 
removing a colleges accreditation entirely, gradated thresholds within quantitative performance 
measures, or multiple years of infractions could be used to determine consequences that stop short of 
complete termination of accreditation. For example, a college in its first year of low graduation rates 


might get a warning, but in a second or third year, or with even lower graduation rates, might see its Title 
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IV funding reduced by 10 percent. Alternatively, low-performing institutions might be subject to a 
different set of rules like enforcing an 80-20 rule at low-performing for-profit institutions rather than a 
90-10 rule. 

One potential challenge to the possibility of sustaining two, non-redundant systems, is that there 
is a risk that one system ultimately “disciplines” the other so that the two converge—effectively 
eliminating the value of one or both systems. Scholars of quantification have long noted that rhetorical 
power of quantitative information—a power that is not necessarily contingent on its technical merit (e.g. 
Carruthers & Espeland, 1991). Especially in instances where professional expertise is weak or professional 
standards are uncertain, the impulse toward numbers-based decision making can be quite strong (e.g. 
Carson, 2007; Porter, 1996). In these instances, emphasizing the plain meaning of numbers can be a way 
of heading off claims of politics or bias but it can also be a way of side-stepping larger questions of 
morality or incommensurable value claims (e.g. Fourcade, 2011). These are not just theoretical concerns: 
to increase pressure on accreditation bodies, ED has moved to create scorecards for accreditation 
agencies with the goal of allowing for direct comparison of accreditors’ performance (Fain, 2016). Given 
that different accreditors serve different constituencies and have different systems of value and 
evaluation, the possibility of an evaluation monoculture developing is real and considerable. 

Future Research 

Given the public availability of both postsecondary accreditation outcomes and a wide array of 
quantitative performance outcomes, there is a promising future research agenda in this area. This work 
can and should continue to make contributions to the literature on the effects of quantification and 
accountability by working to answer questions about how accreditors can (and perhaps do) use 
quantitative performance measures to improve their work, or how institutional behavior shifts in 
response to quantitative accountability pressures. This data can also be used to provide a new look into 


the work of accrediting bodies, providing more detail about how accreditors differ in their standards, or 
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which accreditors are most effective in promoting the performance of their member institutions. Finally, 
future research can work to answer more fundamental questions about how much of the various 
quantitative measures the public cares about are under the direct control of an institution—and thus a 
fair and useful accountability lever—or are instead a consequence of the types of students an institution 
serves, or other situational characteristics of an institution. Related to these issues are questions about 
how to benchmark the performance of colleges, and what colleges it is appropriate to compare a given 
institution to. 
Conclusion 

There is general agreement that the current accreditation system is not currently working to 
address major concerns about the performance of the U.S. higher accreditation system. Accreditors have 
too many different responsibilities and too few tools to secure the outcomes expected by lawmakers in 
the public. Despite the failure of the Obama administration’s proposed college rating system, it is clear 
that pressures to increase postsecondary accountability are not diminishing. It seems equally clear that, 
despite its shortcomings, the accreditation system is unlikely to be eliminated or dramatically 
restructured. The most productive way forward, therefore, is to try to develop new systems that can 
complement the efforts and improve the capacity of accreditors and provide them new tools to 
productively promote improvement in low-performing schools. Our work suggests that neither 
accreditors nor quantitative performance measures on their own are likely to fulfill the nation’s 
accountability needs. Future work should endeavor to find ways of developing stricter quantitative 


scrutiny of colleges while preserving the discretion and expert judgment accreditors bring to their work. . 
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Notes 
1 Downloaded October 1, 2016 from http://ope.ed.gov/accreditation/GetDownLoadFile.aspx. 
2 These rates have been criticized for including only first-time, full-time, degree seeking students and for 
poorly tracking student transfers (U.S. Department of Education, 2014). 
3 Note that the threshold-setting school is not the same for each measure. The lowest performing school 
based on the graduation rate of the schools that lost its accreditation is not the same schools as the 
lowest performing school in terms of default rate. Yet there are colleges that are performing worse than 


both of these schools on both measures. 
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Table 1 


PURPOSES OF ACCREDITATION AND PROPOSED COLLEGE RATING SYSTEM 


Selected Purposes of Accreditation Proposed Purposes of College Rating System 
e “Assist prospective students in identifying e “Help students and families make informed 
acceptable institutions” choices about searching for and selecting a 
college” 
e “Verify that an institution or program meets e “Help colleges and universities measure, 
established standards” benchmark and continue to improve across 
shared principles of access, affordability, and 
outcomes” 
e “Help to identify institutions and programs for e “Enable the incentives and accountability 
the investment of public and private funds” structure in the federal student aid program 
e “Create goals for self-improvement...and to be properly aligned to these key principles” 


stimulate a general raising of standards 
among educational institutions” 


Sources: Hegji (2014), p. 1, U.S. Department of Education (2014), p. 1 
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Table 2 


DESCRIPTIVE STATISTICS FOR INSTITUTIONS, BY ACCREDITATION ACTION AND ACCREDITOR TYPE 


ED Data Lost Accreditation Probation Maintained 
Accreditor Type 

4-year All Regional National All Regional National All Regional National 
Public (%) 0.0 0.0 0.0 3.3 5.1 0.0 14.0 21.6 0.0 
N 0 0 0 2 2 0 761 711 0 
Private (%) 20.6 3h 5.6 32.8 51.3 4.6 30.9 38.5, 18.6 
N 7 6 1 20 20 1 1676 1266 337 
For-profit (%) 5.9 6.3 5.6 8.2 10.3 0.0 14.4 9.9 22.0 
N 2 1 1 5 4 0 780 325 398 

2-year 
Public (%) 20.6 43.8 0.0 26.2 28.2 22.7 19.4 27.6 55 
N 7 7 0 16 11 B; 1050 907 99 
Private (%) 8.8 12.5 5.6 3.3 5.1 0.0 3.1 1.0 5.0 
N 3 2 1 2 2 0 166 32 91 
For-profit (%) 44.1 0.0 83.3 26.2 0.0 72.7 18.3 1.6 48.8 
N 15 0 15 16 0 16 990 51 884 

Enrollment 
Total undergrad enrollment 633.3 931.6 36.8 1361.4 1899.5 456.5 3266.4 4847.9 442.1 
% Pell recipients 50.9 46.2 60.4 54.2 53.9 54.8 49.3 42.0 62.6 
% Black 21.4 15.8 32.5, 23.8 24.1 23.4 17.1 14.7 21.1 
% Hispanic 6.1 47 8.9 14.9 14.8 15.0 14.2 13.5 15.3 
% Asian 0.6 0.7 0.5 3.8 4.3 3.1 3.4 3.8 2.8 

N 18 6 12 61 39 22 4,859 3,292 1,810 

CAP Data Lost Accreditation Major Action Minor Action Maintained 

Accreditor Type 

4-year All Regional National All Regional National All Regional National All Regional National 
Public (%) 0.0 0.0 0.0 6.6 14.1 0.0 10.5 11.1 0.0 10.6 15.7 0.0 
N 0 0 0 14 14 0 22 22 0 42 42 0 
Private (%) 50.0 80.0 0.0 17.5 37.4 0.0 40.7 40.4 45.5 25.7 37.8 0.8 
N 8 8 0 37 37 0 85 80 5 102 101 1 
For-profit (%) 12.5 0.0 10.0 14.7 3.0 25.0 7.2 TA 3.1 12.9 71 24.6 
N 1 0 1 31 3 28 15 14 1 51 19 32 

2-year 
Public (%) 25.0 10.0 0.0 23.2 41.4 TA 37.3 39.4 0.0 25.7 36.7 3.1 
N 2 1 0 49 41 8 78 78 0 102 98.0 4 
Private (%) 20.0 10.0 30.0 43 4.0 4.5 2.4 2.0 9.1 23 2 3.1 
N 4 1 ) 9 4 5 5 4 1 9 5.0 4 
For-profit (%) 12.5 0.0 60.0 33.7 0.0 63.4 1.9 0.0 36.4 22.9 1 68.5, 
N 1 0 6 71 0 71 4 0 4 61 2.0 89 

Enrollment 
Total undergrad enrollment 2316.6 4437.3 195.8 26.6 4606.4 910.9 5664.5 5996.1 178.7 3912.3 5241.9 867.9 
% Pell recipients 54.5 54.7 54.3 59.1 48.5, 68.5, 46.9 45.9 63.9 51.9 47.0 63.0 
% Black 24S 40.8 16.7 25.1 24.2 25.9 24.8 24.7 26.9 23.0 22.5, 24.2 
% Hispanic 5.8 8.3 33 18.8 20.8 16.9 16.2 16.0 19.3 17.0 16.1 12.1 


% Asian 3.7 5.9 14 3.3 4.8 2.0 3.6 3.8 1.6 3.3 3.8 23 
N 20 10 10 211 99 112 209 198 11 397 267 130 
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ED Data Net Price ($) Graduation Rate (%) Grad Rate Relative to Prediction (%) Default Rate (%) Default Rate Relative to Prediction (%) 
4-Year Institutions All Public Private For-profit All Public Private For-profit All Public Private For-profit All Public Private For-profit All Public Private For-profit 
Lost Accreditation 19,022. - 19,088 18,760 37.1 - 42.6 15.0 -12.1 - -4.8 41.2 12.8 - 6.2 26.0 1.7 - -2.5 10.2 
Probation 16,479 10,380 16,755 18,034 310 376 31.1 28.3 -13.2 23.3 2-143 -24.6 13.7 193 134 14.0 3.6 0.6 49 -0.2 
Accredited 18,963 12,629 20,821 22,038 480 435 53.9 36.9 0.7 13.3 -0.3 -12.8 8.9 8.4 6.4 14.8 -0.6 -2.4 0.4 -0.9 

2-year Institutions 
Lost Accreditation 12,038 11,089 1,201 14635 61.5 72.85 74.0 50.0 15.2 36.6 20.0 -3.4 16.1 16.1 - - 2.2 2.2 - - 
Probation 11,025 8,201 7,038 14569 419 248 2.9 65.4 -3.7 -10.5 -47.6 9.3 9.9 17.5 - 11.4 -0.4 51 - -5.0 
Accredited 12,599 7,370 18,614 17,591 435 27.1 56.6 60.3 -0.7 -7.6 3.6 6.6 15.7 18.1 8.4 14.5 0.9 5.3 -1.0 -3.1 
CAP Data Net Price Graduation Rate Grad Rate Relative to Prediction Default Rate Default Rate Relative to Prediction (%) 
4-Year Institutions All Public Private For-profit All Public Private For-profit All Public Private For-profit All Public Private For-profit All Public Private For-profit 
Lost Accreditation 19,679 - 18,677 24,691 30.7 - 29.9 34.2 -14.2 - -12.9 -20.3 11.0 - 10.7 21.1 -0.1 - 1.1 12.0 
Major Action 17,678 8,361 17,789 22,572 35.1 334 318 40.9 -6.6 9.6 -114 -10.4 15.9 94 13.7 21.4 2.4 -6.4 4.0 4.9 
Minor Action 18,023 11,703 19,227 21,999 394 386 42.3 27.1 44 12.3 -5.7 -24.2 121 119 117 14.8 1.3 -2.8 3.1 -1.2 
Accredited 17,344 10,609 19,881 19,713 41.0 395 43.9 37.4 -1.7 15.7 -4.5 -13.2 120 109 97 18.0 0.8 -3.7 23 2.3 
2-year Institutions 
Lost Accreditation 17,212 3,304 19,436 20,830 446 314 403 57.8 -10.3 -20.3. -21.2 11.2 115 21.1 8.3 - 3.6 12.0 0.8 - 
Major Action 14,156 6,223 12,599 20,188 440 246 43.3 61.0 -1.0 -12.6 = -9.0 10.2 194 = 15.3 6.0 23.5 2.2 1.6 -6.7 3.3 
Minor Action 7419 6411 16,114 16,201 23.1 196 448 63.0 -148 -170 = -2.3 12.3 16.1 15.7 169 22.7 2.0 1.8 44 3.8 
Accredited 11,475 6,788 19,111 19,342 37.1 223 483 57.3 -6.7 -140 9 -5.1 3.9 183 168 128 20.7 3.8 45 2.7 29) 
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PERCENT OF FOUR-YEAR ACCREDITED INSTITUTIONS WITH QUANTITATIVE PERFORMANCE LOWER THAN COMPARISON GROUP, 


BY LAST ACCREDITATION ACTION 
ED Data Lowest Performing Average Performance Highest Performing N 
Lost Accreditation All Public Private For-profit All Public Private For-profit All Public Private For-profit 
Net Price 32.1 310 39.1 51.5 510 60 604 69.8 70.4 310 808 91.8 2,556 
Graduation Rate 62 3.1 2.7 1.3 314 33.0 20.5 56.6 70.1 77.0 80.2 86.8 2,453 
Grad Rate Relative to Prediction 2.7 0.0 1.0 1.5 238 5.9 214 55.7 56.9 285 60.7 83.4 2,452 
Default Rate 12 0.9 1.3 1.3 273 18.1 = 23.2 68.2 62.9 69.7 47.2 89.3 2,872 
Default Rate Relative to Prediction 2.9 2.4 3.9 1.5 22.9 15.1 23.5 29.7 79.7 55.7 94.3 76.5 2,600 
Probation 
Net Price 204 04 278 26.8 63.6 186 75.8 87.6 99.9 99.6 100.0 100.0 2,556 
Graduation Rate 2.7 08 2:7 5.3 215 206 13.5 41.7 864 923 80.2 93.8 2,453 
Grad Rate Relative to Prediction 0.7 = =0.0 1.0 0.9 22.7 5.0 19.0 53.8 91.8 808 96.1 95.3 2,452 
Default Rate 0.1 0.0 0.1 0.3 22.3 16.1 9.5 55.5 83.3 92.1 73.9 94.3 2,872 
Default Rate Relative to Prediction 0.1 0.0 0.1 0.2 13.7 2 154 13.1 99.2 99.6 99.9 97.6 2,600 
CAP Data 
Lost Accreditation 
Net Price 14.1 24 18.4 18.4 32.1 24 448 39.5 44 310 908 89.5 156 
Graduation Rate TA 71 6.6 8.9 44.8 524 408 44.4 86.5 88.1 80.3 95.6 163 
Grad Rate Relative to Prediction 18 0.0 1.3 44 319 7.2 31.6 55.6 65.0 214 77.6 84.4 163 
Default Rate 35.1 33.3 20.4 69.1 47.7 462 344 78.6 64.6 718 505 90.5 174 
Default Rate Relative to Prediction 196 26 20.5 34.1 50.0 154 60.3 63.4 80.4 53.8 94.9 78.0 158 
Major Action 
Net Price 19 0.0 2.6 2.6 494 71 63.2 684 100.0 100.0 100.0 100.0 156 
Graduation Rate 18 0.0 1.3 44 28.2 23.8 = 23.7 40.0 92.0 90.5 89.5 97.8 163 
Grad Rate Relative to Prediction 0.6 0.0 0.0 2.2 30.1 7.1 28.9 53.3 95.1 88.1 97.4 97.8 163 
Default Rate 0.0 0.0 0.0 0.0 25:3). Te. 17.2 59.5 97.1 100.0 94.6 100.0 174 
Default Rate Relative to Prediction 0.6 0.0 1.3 0.0 373 103 41.0 56.1 100.0 100.0 100.0 100.0 158 
Minor Action 
Net Price 19 0.0 2.6 2.6 455 24 59.2 65.8 96.2 88.1 100.0 97.4 156 
Graduation Rate 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 163 
Grad Rate Relative to Prediction 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 163 
Default Rate 0.0 0.0 0.0 0.0 43.1 385 30.1 76.2 97.1 100.0 94.6 97.6 174 
Default Rate Relative to Prediction 0.0 0.0 0.0 0.0 39.9 103 44.9 58.5 99.4 100.0 100.0 97.6 158 
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PERCENT OF TWO-YEAR ACCREDITED INSTITUTIONS WITH QUANTITATIVE PERFORMANCE LOWER THAN COMPARISON GROUP, 


BY LAST ACCREDITATION ACTION 
ED Data Lowest Performing Average Performance Highest Pefrorming N 
Lost Accreditation All Public Private For-profit All Public Private For-profit All Public Private For-profit 
Net Price 95 02 26.1 17.6 22.7 0.2 47.1 44.5 35.3' 25 68.9 67.1 2,019 
Graduation Rate 13.7 254 6.7 1.1 49.4 83.7 27.5 13.2 67.6 90.9 508 43.2 2,007 
Grad Rate Relative to Prediction 0.8 0.0 5.8 1.0 34.7 534 26.7 14.4 640 85.0 47.5 42.3 2,006 
Default Rate 6.7 89.6 3.1 4.6 644 80.1 164 56.5, 92.8 98.7 60.2 92.1 1,793 
Default Rate Relative to Prediction 11.2 19.5 3.9 4.3 43.9 70.8 25.8 20.7 722.9 945 64.1 53.4 1,789 
Probation 
Net Price 5.1 0.0 16.0 9.3 30.7 13 58.8 59.4 98.8 97.7 58.8 99.8 2,019 
Graduation Rate 2.9 49 42 0.6 43.0 75.9 21.7 8.3 83.2 94.0 68.3 72.8 2,007 
Grad Rate Relative to Prediction 04 00 2.5 0.6 32.1 48.9 26.7 13.5 89.6 91.5 81.7 88.5 2,006 
Default Rate 1517 0.0 14 58.7 725 148 52.1 97.5 99.5 82.0 98.0 1,793 
Default Rate Relative to Prediction 10 17 0.8 0.4 348 58.9 15.6 14.5 94.2 100.0 100.0 87.7 1,789 
CAP Data 
Lost Accreditation 
Net Price 123 0.0 25.0 9.4 36.2 0.0 25.0 34.0 93.9 10 87.5 96.2 163 
Graduation Rate 144 245 125 0.0 58.3 59.1 37.5 71 794 98.0 62.5 54.3 180 
Grad Rate Relative to Prediction 0.1 0.0 12.5 0.0 44.1 68.6 25.0 10.1 79.3 96.1 62.5 56.5, 179 
Default Rate 70.8 67.9 33.3 771.6 87.0 864 50.0 91.0 98.1 97.5 100.0 98.5, 154 
Default Rate Relative to Prediction 38.6 35.8 33.3 42.4 765 81.5 83.3 69.7 85.6 95.1 83.3 74.2 153 
Major Action 
Net Price 18 00 25.0 3.8 16.6 0.0 100.0 56.6 994 99.0 100.0 100.0 163 
Graduation Rate 44 69 12.5 0.0 50.0 83.3 25.0 43 93.9 99.0 87.5 87.1 180 
Grad Rate Relative to Prediction 0.0 0.0 0.0 0.0 38.0 59.8 25.0 dD 96.6 98.0 100.0 94.2 179 
Default Rate 0.7 12 0.0 0.0 60.4 53.1 16.7 73.1 994 98.8 100.0 100.0 154 
Default Rate Relative to Prediction 0.7 «12 0.0 0.0 62.1 61.7 66.7 62.1 100.0 100.0 100.0 100.0 153 
Minor Action 
Net Price 18 00 12.5 3.8 19.0 0.0 25.0 54.7 92.0 87.3 100.0 100.0 163 
Graduation Rate 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 180 
Grad Rate Relative to Prediction 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 179 
Default Rate 0.0 0.0 0.0 0.0 79.2 76.5 50.0 85.1 99.4 98.8 100.0 100.0 154 
Default Rate Relative to Prediction 0.0 0.0 0.0 0.0 69.3 71.6 66.7 66.7 98.0 98.8 100.0 97.0 153 
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Table 6 
GRADUATION AND DEFAULT RATE PERFORMANCE RELATIVE TO REGRESSION-ADJUSTED EXPECTATION, 


BY LAST ACCREDITATION ACTION, LEVEL, AND CONTROL 


Graduation Rate 
ED Data Percent in Bottom 25% vs. Predicted Percent Below Predicted Percent in Top 25% vs. Predicted 
4-Year 2-Year 4-Year 2-Year 4-Year 2-Year 
All Public Private For: All Public Private For: All Public Private For. All Public Private For: All Public Private For All Public Private For: 
profit profit profit profit profit profit 
Lost Accreditation 40.0 - 25.0 100.0 16.7 0.0 0.0 33.3 80.0 - 75.0 100.0 33.3 0.0 0.0 66.7 0.0 - 0.0 0.0 66.7 100.0 100.0 33.3 
Probation 56.5 0.0 56.3 0.8 394 500 1000 20.0 826 0.0 93.8 0.8 63.6 81.3 1000 400 174 1.0 6.3 0.2 364 18.8 1.0 60.0 
Accredited 24.9 6.5 216 55.7 236 347 250 108 479 195 50.0 78.7 57.7 806 45.0 333 25.8 49.8 20.5 71.9 25.5 12.0 43.3 38.5 
CAP Data 
4-Year 2-Year 4-Year 2-Year 4-Year 2-Year 
All Public Private Bor: All Public Private or All Public Private Bor All Public Private jie All Public Private For All Public Private For: 
Rae ret ee pitt. A poh a ee pret, a ES ee hohe Ca pot, erp 
Lost Accreditation 50.0 - 40.0 100.0 50.0 1000 66.7 0.0 ~~ 100.0 - 100.0 100.0 50.0 1000 66.7 0.0 0.0 - 0.0 0.0 16.7 0.0 0.0 50.0 
Major Action 45.5 143 517 565 250 49.0 42.9 1.8 63.6 286 75.9 696 527 85.7 714 214 182 50.0 6.9 13.0 286 10.2 285 446 
Minor Action 32.4 9.1 28.8 85.7 506 564 0.0 0.0 60.8 22.7 68.2 85.7 920 962 800 250 147 45.5 6.1 7A 3.4 1.3 20.0 25.0 
Accredited 31.9 7A 316 556 256 39.2 25.0 at 27.1 143 658 82.2 711 951 625 37.1 215 59.5 10.5 44 11.1 2.9 25.0 214 
Default Rate 
ED Data Percent in Bottom 25% vs. Predicted Percent Below Predicted Percent in Top 25% vs. Predicted 
4-Year 2-Year 4-Year 2-Year 4-Year 2-Year 
All Public Private For: All Public Private For: All Public Private Bor All Public Private — All Public Private For: All Public Private For: 
profit profit profit profit profit profit 
Lost Accreditation 33.3: - 50.0 0.0 20.0 20.0 - - 66.7 - 100.0 0.0 20.0 20.0 - - 33:3 - 0.0 1000 200 20.0 - - 
Probation 45 0.0 0.0 25.0 27.3 0.0 - 50.0 27.3 0.0 235 50.0 545 20.0 - 83.3 545 0.0 58.8 50.0 273 40.0 - 16.7 
Accredited 24.8 52.7 9.2 27.0 26.2 47 344 456 567 75.2 S507 S511 466 190 656 70.2 30.5 175 351 340 230 103 10.1 8.4 
CAP Data 
4-Year 2-Year 4-Year 2-Year 4-Year 2-Year 
All Public Private For: All Public Private For: All Public Private For: All Public Private For: All Public Private For: All Public Private For: 
profit profit profit profit profit profit 
Lost Accreditation 333 - 50.0 0.0 0.0 0.0 0.0 - 66.7 - 50.0 100.0 25.0 0.0 33.3 - 33.3 - 50.0 0.0 25.0 1000 0.0 - 
Major Action 18.9 69.2 3.0 143 221 304 800 113 365 846 303 214 363 39.1 800 306 541 77 60.6 679 37.2 45.7 200 32.3 
Minor Action 19.7 47.6 110 286 205 21.1 0.0 25.0 436 714 31.7 714 27.7 276 0.0 50.0 48.7 238 585 286 386 395 33.3 20.0 


Accredited 195 53.8 43 214 13.6 49 16.7 239 466 846 355 35.7 247 185 333 31.3 443 10.3 51.6 595 35.1 35.8 16.7 35.8 
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Figure 1. Accredited Four-Year College Graduation and Default Rate Performance. Lines indicate the 


performance on each outcome of the lowest, average, and highest performing institutions that lost their 


accreditation. 
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Figure 2. Accredited Two-Year College Graduation and Default Rate Performance. Lines indicate the 


performance on each outcome of the lowest, average, and highest performing institutions that lost their 


accreditation. 
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Figure Al. Accredited Four-Year College Graduation and Default Rate Performance, CAP Data 


Lines indicate the performance on each outcome of the lowest, average, and highest performing 


institutions that lost their accreditation. 
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Figure A2. Accredited Two-Year College Graduation and Default Rate Performance, CAP Data. 


Lines indicate the performance on each outcome of the lowest, average, and highest performing 


institutions that lost their accreditation. 


